Video Conferencing

ABSTRACT

A video conferencing system is provided, in which at least two cameras are used to capture images of people at a first location participating in a video conference. One or more active speakers are identified among the people at the location, and one of the at least two cameras is automatically selected based on a position or positions of the one or more active speakers. Images from the selected camera are provided to a person at a second location participating the video conference.

RELATED APPLICATIONS

This application is a continuation of and claims the benefit of U.S.patent application Ser. No. 11/966,674, filed on Dec. 28, 2007, whichclaims priority to U.S. Provisional Patent Application No. 60/877,288,filed on Dec. 28, 2006. The above applications are incorporated byreference in their entirety.

BACKGROUND

This invention relates to video conferencing.

Video conferencing allows groups of people separated by large distancesto have conferences and meetings. In some examples, two parties of avideo conference each uses a video conferencing system that includes acamera for capturing images of local participants and a display forshowing images of remote participants (and optionally the localparticipants) of the video conference. The participants may manuallycontrol the cameras to adjust zoom and viewing angle in order to clearlyshow the faces of the speakers during the conference. In some examples,a video conferencing system may include an array of microphones todetect sound sources using triangulation, and automatically direct thecamera to zoom in on the speaker.

SUMMARY

In one aspect, in general, a method of conducting a video conference isprovided, in which at least two cameras are used to capture images ofpeople at a first location participating in a video conference; one ormore active speakers are identified among the people at the location;one of the at least two cameras is automatically selected based on aposition or positions of the one or more active speakers; and imagesfrom the selected camera are provided to a person at a second locationparticipating the video conference.

Implementations may include one or more of the following features. Theimages are optically or digitally zoomed based on the one or more activespeakers. Identifying one or more active speakers includes identifyingpeople who have spoken within a predetermined period of time. A databaseof one or more active speakers is maintained, adding a person who startsto speak to the database, and removing a person who has not spoken for apredetermined period of time from the database. A user interface isprovided to allow adjustment of the duration of the predetermined periodof time.

In another aspect, in general, at least two cameras are used to captureimages of people at a location; one or more active speakers at thelocation are identified; one of at least two cameras are automaticallyselected based on a position or positions of the one or more activespeakers; and images from the selected camera are provided.

Implementations may include one or more of the following features.Identifying one or more active speakers includes identifying people whohave spoken within a predetermined period of time. A database of one ormore active speakers is maintained, a person who starts to speak isadded to the database, and a person who has not spoken for apredetermined period of time is removed from the database. The databaseis periodically updated and the selection of camera is automaticallyadjusted based on the updated database. Maintaining the database of oneor more active speakers includes storing information about when eachspeaker starts and ends speaking Maintaining the database of one or moreactive speakers includes storing information about a coordinate of eachactive speaker.

Selecting one of at least two cameras includes selecting one of thecameras having a smallest view offset angle with respect to the one ormore active speakers. The images are sent to a remote party who isconducting a video conference with the people at the location. Theposition or positions of the one or more active speakers are determined.Determining positions of the active speakers includes determiningpositions of the active speakers by triangulation. Determining positionsof the active speakers by triangulation includes triangulation based onsignals from a microphone array. The camera is automatically zoomed tomore clearly show the one or more active speakers. A zoom value isdetermined based on a distance or distances between the camera and theone or more active speakers. Determining the zoom value includesdetermining a zoom value to provide a first margin between a referencepoint of a left-most active speaker and a left border of the image, anda second margin between a reference point of a right-most active speakerand a right border of the image. A zoom value is determined based on adistance between the camera and a closest of the one or more activespeakers. A viewing angle of the camera is automatically adjusted tomore clearly show the one or more active speakers.

In another aspect, in general, a log of one or more active speakers at alocation is maintained; a zoom factor and a viewing direction of acamera are automatically determined based on a position or positions ofthe one or more active speakers such that the active speakers are withina viewing range of the camera; and images of the one or more activespeakers are provided.

Implementations may include one or more of the following features. Theone or more active speakers are periodically identified and the log isupdated to include the identified one or more active speakers. Updatingthe log includes adding a person who starts to speak to the log andremoving a person who has not spoken for a predetermined period of timefrom the log. Determining the zoom factor and viewing direction of thecamera includes determining a zoom factor and viewing direction toprovide a first margin between a reference point of a left-most activespeaker and left borders of the images, and a second margin between areference point of a right-most active speaker and right borders of theimages.

In another aspect, in general, active speakers in a room are identified,the room having cameras for capturing images of people in the room; asubset of less than all of the cameras in the room is selected; andimages from the selected subset of cameras are provided to show theactive speakers. Identifying active speakers includes identifying peoplein the room who have spoken within a predetermined period of time.

In another aspect, in general, a video conferencing system is provided.At least two cameras capture images of people at a first locationparticipating a video conference; a speaker identifier identifies one ormore active speakers at the first location; a data processor selects oneof the at least two cameras based on a position or positions of the oneor more active speakers and provides images from the selected camera toshow the one or more active speakers; and a communication interfacesends the images to a person at a second location participating thevideo conference.

Implementations may include one or more of the following features. Amicrophone array identifies speakers in the room based on triangulation.A storage stores information about speakers who have spoken within apredetermined period of time.

In another aspect, in general, at least two cameras capture images ofpeople at a location; and a data processor selects one of the at leasttwo cameras based on a position or positions of one or more activespeakers at the location and provide images from the selected camera toshow the one or more active speakers.

Implementations may include one or more of the following features. Aspeaker identifier identifies active speakers in the room. The speakeridentifier identifies active speakers in the room by identifying one ormore people who have spoken within a predetermined period of time. Thespeaker identifier includes a microphone array that enablesdetermination of positions of the active speakers by triangulation. Astorage stores coordinates of the active speakers in the room, and timepoints when each active speaker started and ended talking The dataprocessor selects one of the at least two cameras by selecting thecamera having a smallest view offset angle with respect to the activespeakers. The data processor executes a video conferencing process tosend the image to a remote party who is video conferencing with thepeople at the location. The data processor controls a zoom factor and aviewing direction of the camera such that the active speakers are withina viewing range of the camera.

In another aspect, in general, at least two cameras capture images ofpeople in a room; a speaker identifier identifies active speakers in theroom; and a data processor selects a subset of the at least two camerasand provide at least one image from the subset of the at least twocameras to show the active speakers.

These and other aspects and features, and combinations of them, may beexpressed as methods, apparatus, systems, means for performingfunctions, computer program products, and in other ways.

The apparatuses and methods can have one or more of the followingadvantages. Interpersonal dynamics can be displayed with the videoconferencing system. By using multiple cameras, the viewing angle of thepeople speaking can be improved, when there are several participants inthe video conference, most or all of the speakers do not have to turntheir heads significantly in order to face one of the cameras. Thesystem can automatically choose a camera and its viewing direction andzoom factor based on the person or persons speaking so that if there aretwo or more people conducting a conversation, images of the two or morepeople can all be captured by the camera. Effectiveness of videoconferences can be increased. Use of digital zooming can reducemechanical complexity and reduce the response delay caused by motioncontrollers.

Other features and advantages of the invention are apparent from thefollowing description, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1A is a diagram of an environment in which a conversation sensitivevideo conferencing system can be used.

FIG. 1B is a diagram of a conversation sensitive video conferencingsystem.

FIG. 2 is a block diagram of a video camera.

FIG. 3 is a block diagram of a video conferencing transceiver.

FIG. 4 is a diagram of a memory map.

FIG. 5 shows software classes for implementing a conversation sensitivevideo conferencing system.

FIG. 6 is a diagram showing a sequence of events and interactionsbetween objects when the conversation sensitive video conferencingsystem is used to conduct a video conference.

FIG. 7 is a flow diagram of a process implemented by anAcknowledgeTalker( ) method.

FIG. 8A is a flow diagram of a process implemented by an AddTalker( )method.

FIG. 8B is a flow diagram of a process implemented by a RemoveTalker( )method.

FIG. 9 is a diagram of relative positions of a speaker and cameras.

FIG. 10A is a diagram of a source map.

FIG. 10B is a diagram of a talker map.

FIG. 11 is a flow diagram of a conversation sensitive zoom process.

FIG. 12 is a diagram showing positions of active and inactive speakersand the left and right display boundaries.

FIG. 13 shows a diagram for determining a horizontal position (xPos) anda vertical position (yPos) of a selected scene to achieve a particulardigital zoom effect.

FIGS. 14A to FIG. 14D are images of people talking in a room.

DESCRIPTION

Referring to FIG. 1A, an example of a conversation sensitive videoconferencing system includes multiple video cameras 103 to capture avideo or images of participants of a video conference from variousviewing angles. The video conferencing system automatically selects oneof the cameras 103 to capture images of people who have spoken within apredetermined period of time. When two or more people are in aconversation or discussion, the viewing direction and zoom factor of theselected camera are automatically adjusted so that images captured bythe selected camera show most or all of the people activelyparticipating in the conversation. When additional people join in theconversation, or when some people drop out of the conversation, thechoice of camera and the viewing direction and zoom factor of theselected camera are automatically re-adjusted so that images captured bythe selected camera show most or all of the people currentlyparticipating in the conversation.

An advantage of the conversation sensitive video conferencing system isthat remote participants of the video conference can see more clearlythe people who are actively participating in the conversation. If onlyone camera were used, it may be difficult to provide a good viewingangle for all or most of the participants. Some participants may havetheir backs toward the camera and would have to turn their headssignificantly in order to face the camera. If the viewing angle and zoomfactor of the camera were fixed during the conference so that the cameracapture images showing all of the people in the conference room, thefaces of some of the people may be small, and it may be difficult forthe remote participants to see clearly the people who are speaking

Another advantage of the conversation sensitive video conferencingsystem is that it is not necessary to manually select one of the camerasor adjust the viewing angle and zoom factor to capture images of peopleactively participating in the conversation. Participants of the videoconference can focus on the discussion, rather than being distracted bythe need for constant adjustment of the cameras.

Each of the video cameras 103 is capable of capturing a video thatincludes a sequence of images. In this description, the images capturedby the camera can be either still images or a sequence of images thatform a video.

Referring to FIG. 1B, an example of a conversation sensitive videoconferencing system 100 can be used to show interpersonal dynamics amonglocal participants of a video conference. The system 100 includes acamera assembly 102 having multiple cameras (e.g., 103 a and 103 b,collectively referenced as 103) that capture images of the peopleparticipating in the video conference from various viewing angles. Avideo conferencing transceiver (VCT) 104 controls the camera assembly102 to select one of the cameras to capture images of people who havespoken within a predetermined period of time. When two or more peopleare actively participating in a discussion, the video conferencingtransceiver 104 automatically adjusts the viewing direction and zoomfactor of the selected camera so that images captured by the selectedcamera show most or all of the people actively participating in thediscussion. This is better than displaying images showing allparticipants of the video conference (where each participant's face maybe small and not clearly visible) or displaying images of individualspeakers where the images switch from one speaker to another.

A conference display 106 is provided so that local participants of thevideo conference can see the images are captured by the selected camera103, as well as images of the remote participants of the videoconference. The video conferencing transceiver 104 is connected to theremote site through, for example, a broadband connection 108. A userkeypad 130 is provided to allow local participants of the videconference to control the video conferencing transceiver 104 to varioussystem settings and parameters.

In some implementations, the system 100 includes a programming interfaceto allow configurations of the video conferencing transceiver 104 to beupdated using, for example, a personal computer. A speaker locationdetector 112 determines the locations of speakers. The speaker locationdetector 112 may include, for example, an array of microphones 113 todetect utterances from a speaker and determine the location of thespeaker based on triangulation.

The camera assembly 102 sends audio signals picked up by the microphones113 to the video conferencing transceiver 104 through signal lines 114.The camera assembly 102 sends video signals to the video conferencingtransceiver 104 through signal lines 116, which can be, e.g., IEEE 1394cables. The signal lines 116 also transmit control signals from thevideo conferencing transceiver 104 to the camera assembly 102. Thecamera assembly 102 sends control signals for generating chirp signalsfor use in self-calibration to the video conferencing transceiver 104through signal lines 124. The video conferencing transceiver 104transmits VGA signals 118 to the programming interface 110, and receivesmouse data 126 and keyboard data 128 from the programming interface 110.The video conferencing transceiver 104 sends video signals and audiosignals to the conference display 106 through a VGA cable 120 and anaudio cable 122, respectively.

Referring to FIG. 2, in some implementations, each video camera 103 inthe video camera assembly 102 includes a sensor 140 for capturingimages, a camera microphone 142 for capturing audio signals, and aninput/output interface 144 (e.g., an IEEE 1394 interface) forinterfacing with the video conferencing transceiver 104. The sensor 140can be, e.g., a charge coupled device (CCD) sensor or a complimentarymetal oxide semiconductor (CMOS) sensor. The video camera 103 is coupledto a chirp generator 146 that is used to generate chirp signals for usein self calibrating the system 100.

Referring to FIG. 3, in some implementations, the video conferencingtransceiver 104 includes a motherboard 150 hosting a central processingunit (CPU) 152, memory devices 154, and a chipset 156 that controlsvarious input/output devices and storage devices. The CPU 152 can be anytype of microprocessor or microcontroller. The memory devices 154 canbe, for example, dynamic random access memory (DRAM), Flash memory, orother types of memory. The motherboard 150 includes a line out port 170for outputting conference audio signals to the conference display 106. Amicrophone input port 172 is provided to receive audio signals from themicrophones 113 of the speaker location detector 112.

An IEEE 1394 controller 158 is provided to process signals sent throughthe IEEE 1394 bus 116. Sound cards 160 are provided to process audiosignals from the video camera microphones 142. A network interface 162is provided to connect to the broadband connection 108. A video card 164is provided to generate video signals that are sent to the videoconferencing display 106 and to the programming interface 110. A harddrive 166 and an optical disc drive 168 provide mass storage capability.For example, the hard drive 166 can store software programs used tocontrol the system 100 and data generated when running the system 100.

Optionally, the motherboard 150 includes circuitry 174 for processingcalibration control signals for chirp generators.

Referring to FIG. 4, in some implementations, the memory devices 154store various information, including programming instructions 180 forcontrolling the system 100 and constant values 182 that are used by theprogramming instructions 180. The memory devices 154 store variablevalues 184 and stacks of data 186 that are generated and used duringoperation of the system 100. The memory 154 includes a region forstoring output data 192, and a region for storing text 194 to bedisplayed on the conference display 106.

A first video scratch memory 188 is provided to store image data from afirst video camera 103 a, and a second video scratch memory 190 isprovided to store image data from a second video camera 103 b. If morevideo cameras 102 are used, additional scratch memory can be providedfor each video camera 103. Each video scratch memory corresponds to awindow showing images captured by a corresponding video camera 103. Thecamera 103 that is chosen has its window moved to the front of thedisplay screen, and the camera 103 not chosen has its window sent to theback of the display screen. By switching between the video scratchmemory 188 and 190, the system 100 can quickly switch from images fromone video camera 103 a to images from another video camera 103 b.

Referring to FIG. 5, in some implementations, the conversation sensitivevideo conferencing system 100 executes software programs written inobject oriented programming, in which various software classes aredefined. For example, a Camera class 200, a Controller class 202, anAuditor class 204, a Talker class 206, a Meeting Focus class 208, aCommunicator class 210, and a Display class 212 are defined. In thisdescription, the same reference number is used for a class and objectsthat belong to the class. For example, the reference number 200 is usedfor both the Camera class and a Camera object belonging to the Cameraclass.

A Camera object 200, which is an instance of the Camera class 200, canbe used for controlling various aspects of one of the cameras 103. TheCamera object 200 can call LibDC 1394 functions to control the videocamera 103 to capture video. The LibDC1394 is a library that provides ahigh level programming interface for controlling IEEE 1394 based camerasthat conform to the 1394-based Digital Camera Specifications. Thelibrary allows control of the camera, including turning the camera on oroff, and have continuous live feed. The Camera object 200 can also callJava media framework (JMF) application programming interface (API).

For example, the Camera object 200 can have methods (or interfaces) suchas StartCapture( ) and StopCapture( ) which can be used for starting orstopping the recording of video data.

The Camera object 200 has a number of parameters that can be adjusted,such as current angle and zoom factor. Adjusting the current angle andzoom factor of the Camera object 200 causes the current angle and zoomfactor of the corresponding camera 103 to be adjusted.

When the video conferencing system 100 is turned on, a Controller object202 (which is an instance of the Controller class 202) starts up a userinterface to allow the user to control the video conferencing system.

For example, the Controller object 202 can have two methods, includingStartConference( ) and StopConference( ) methods, which can be used tostart or stop the system 100 when either a local user or a remote userinitializes or ends a conference.

An Auditor object 204 (which is an instance of the Auditor class 204)can be used to monitor audio signals. For example, there can be threeAuditors 204 each representing one of the microphones 113. If thespeaker position detector 112 includes more than three microphones 113,an additional Auditor object 204 can be provided for each additionalmicrophone 113.

The Auditor object 204 can have a Listen( ) method for reading audiodata from a circular buffer and determining whether the audio datarepresents background noise or speaker utterance. When a speakerutterance is detected, referred to as an “onset”, the Auditor object 204informs a Talker object 206 (described below), which then correlates thesignals from the three Auditor objects 204 to determine the position ofthe speaker.

The Talker object 206 (which is an instance of the Talker class 206) canreceive information provided by the Auditor objects 204 and can have aAcknowledgeTalker( ) method that is used to calculate the location ofthe sound source (e.g., location of the speaker) using correlation andtriangulation. The Talker object 206 also checks to see if the speakerstops speaking, as this information is useful to a conversation sensingalgorithm in determining which persons are still participating in aconversation.

A Meeting Focus object 208 (which is an instance of the Meeting Focusclass 208) implements a conversation sensing algorithm that determineswhich persons (referred to as speakers or talkers) are participating ina conversation, and constructs a talker map 350 (FIG. 10B) that includesinformation about the speakers, the locations of the speakers, and thestart and end times of talk for each speaker. For example, the talkermap 350 can be implemented as a database or a table.

For example, the Meeting Focus object 208 can have two methods,including AddTalker( ) and RemoveTalker( ) methods that are used to addor remove speakers from the talker map.

The Meeting Focus object 208 determines the best view of the speaker(s),taking into account variables such as the camera resolution, dimensionsof the window showing the images of the speakers, percentage of entirescreen being occupied by the images, and centroid of images, etc. TheMeeting Focus object 208 sends this information to a Display object 212(described below).

A Communicator object 210 (which is an instance of the Communicatorclass 210) controls the communications infrastructure of the system 100.If a connection with a remote party is made or broken, the Communicatorobject 210 informs the Controller object 202.

For example, the Communicator object 210 can have three methods,including StartComm( ) method for initiating the communicationinterfaces, a MakeConnection( ) method for establishing a connectionwith a remote party, and a BreakConnection( ) method for ending theconnection with the remote party.

In some implementations, a digital zoom is used. A Display object 212(which is an instance of the Display class 212) calculates a zoom factorbased on requested coordinates and the percentage of the screen that theimages is to be displayed, which are provided by the Meeting Focusobject 208.

For example, the Display object 212 can have three methods, including aDisplayDefault( ) method for displaying images using default parameters,a DisplayScene(0 method for display images that mainly show the activeparticipants of a conversation, and a StopDisplay( ) method that stopsdisplay images that mainly show the active participants of aconversation.

FIG. 6 is a sequence diagram 220 showing a sequence of events andinteractions between objects when the conversation sensitive videoconferencing system 100 is used to conduct a video conference. Thesystem 100 is started by a Controller object 202, which initializes auser interface so that a meeting can be started by a local user 244. TheController object 202 can be invoked by a local user 244, or by a remoteuser that makes an incoming call to invoke a MakeConnection( ) method222 of the Communicator object 210.

The Controller 202 invokes a StartConference( ) method 224, whichinitiates a self-calibration process. The Controller 202 starts buzzersby making a call to the Camera objects 200. During the self-calibrationprocess, the acoustics characteristics of the video conference room isanalyzed and the locations of the cameras are determined. Upon thecompletion of the self-calibration process, the video conference canbegin.

When the Controller 202 finishes self calibration, the Controller 202invokes a StartCapture( ) method 226 of the Camera object 200 toinitialize each of the video cameras 103. The Controller 202 passes aCam parameter to the StartCapture( ) method 226 to indicate which camera103 is to start capturing images. The Camera object 200 invokes aStreamIn( ) method 227 to start a video daemon 243 to enable video datato be written into the memory 154. The Camera object 200 passes an Addrparameter 229 to the StreamIn( ) method 227 to indicate the address ofthe memory 154 to which the video data is to be written. The video inputdaemon 243 sends a CamReady acknowledgement signal 231 to the Controller202 when the camera 103 is ready to stream video data to the memory 154.

The Controller 202 invokes a DisplayDefault( ) method 234 of the Displayobject 212 to display images captured by the camera 103 using defaultdisplay settings. The Controller 202 passes a Cam parameter 235 to theDisplayDefault( ) method 234 to indicate that the images from the camera103 represented by the Cam parameter 235 is to be displayed.

The Display 212 invokes a StreamOut( ) method 240 to start a videooutput daemon 242 so that a video with default settings is shown on theconference display 106 and sent to the remote users of the videoconference.

The Controller 200 invokes a Listen( ) method 228 of an Auditor object204. When the Auditor 204 hears an onset, the Auditor 204 invokes anAcknowledgeTalker( ) method 230 of a Talker object 206 to calculate thelocation of the audio source (i.e., the location of a speaker who juststarted speaking) through trigonometry and calculus algorithms. TheAuditor 204 passes a t parameter 233 to the AcknowledgeTalker( ) method230 to indicate the time of onset.

The Talker 206 invokes an AddTalker( ) method 232 of a Meeting Focusobject 208 to add the new speaker to the talker map. The Talker 206passes a Location parameter 237 to the AddTalker( ) method 232 toindicate the location of the new speaker. The Meeting Focus 208identifies the correct camera that can capture images that include theactive speakers in the talker map. The Meeting Focus object 208determines a center of the image and the size of the image to display.

The Meeting Focus object 208 invokes a DisplayScene( ) method 236 of theDisplay object 212, which determines how pixel values are interpolatedin order to achieve a certain digital zoom. The Meeting Focus object 208passes Cam, Center, and Size parameters 239 to the DisplayScene( )method 236, where Cam represents the selected camera 103, Centerrepresents the location of a center of the portion of image captured bythe video camera 103 to be displayed, and Size represents a size of theportion of image to be displayed. The DisplayScene( ) method 236 causesan image to be shown on the conference display 106, in which the imagehas the proper zoom and is centered near the centroid of the activespeakers.

When there are more than one speaker in the talker map, the MeetingFocus object 208 determines a centroid of the speaker locations andadjusts the image so that the centroid falls near the center of theimage. The Meeting Focus object 208 also determines a zoom factor thataffects the percentage of the screen that is occupied by the speaker(s).For example, if there is only one speaker, a zoom factor may be selectedso that the speaker occupies 30% to 50% of the width of the image. Whenthere are two speakers, a zoom factor may be selected so that thespeakers occupy 50% to 70% of the width of the image, when there arethree or more speakers, a zoom factor may be selected so that thespeakers occupy 70% to 90% of the width of the image, etc. Thepercentage values above are examples only, other values may also beused.

The zoom factor that results in the speaker(s) occupying a particularpercentage of the image can be determined based on a function of thedistance(s) between the camera and the speaker(s). The zoom factor canalso be looked up from a table that has different zoom factors fordifferent camera-speaker distances and different percentage values(representing the percentage that the speakers occupy the image).

When the Talker 206 determines that a speaker has stopped talking for apredetermined period of time, the Talker 206 invokes a RemoveTalker( )method 238 of the Meeting Focus object 208 to remove the speaker fromthe talker map. The Meeting Focus object 208 selects a camera that cancapture images that include the remaining speakers in the talker map.The Meeting Focus object 208 determines the centroid of the speakerlocations and the percentage of screen that is occupied by the remainingspeakers. The Meeting Focus object 208 invokes the DisplayScene( )method 236 to show images having the proper zoom and centered near thecentroid of the remaining active speakers.

The Auditor 204 continues to listen to the signals from the microphones113 of the speaker location detector 112. When the Auditor 204 hears anonset, the Auditor 204 invokes an AcknowledgeTalker( ) method 230,repeating the steps described above for adding a speaker to the talkermap and later removing the speaker from the talker map when the speakerceases speaking Whenever a speaker is added to or removed from thetalker map, the selection of camera and the viewing direction and zoomfactor of the selected camera are adjusted to properly show the currentparticipants of the conversation or discussion.

The video conference can be ended by invoking a StopConference( ) method246 of the Controller object 202. The StopConference( ) method 246 canbe invoked by the local user 244 or by the remote party hanging up thecall, which invokes a BreakConnnection( ) method 242 of the Communicator210. When the StopConference( ) method 246 is invoked, the Controller202 terminates the Listen( ) method 228 and invokes a StopCapture( )method 248 of the Camera object 200 to cause the camera 103 to stopcapturing images. The Controller 202 invokes a StopDisplay( ) method 250of the Display object 212 to end the video output to the conferencedisplay 106.

The following is a description of the various methods of the objectsused in the system 100.

The methods associated with the Camera object 200 include StartCapture() 226 and StopCapture( )248. The StartCapture( ) method 226 causes thesensor 140 of the camera 103 to start operating, and invokes aStreamIn(Addr) method that causes video data to be streamed into thememory 154 starting at address Addr.

The StopCapture( ) method 248 ends the StreamIn(Addr) method to end thestreaming of video data to the memory 154.

The methods associated with the Controller object 202 include Main( )StartConference( ) 224, and StopConference( ) 246 methods. The Main( )method is used to set up a window for showing the video images of thevideo conference, invoke an InitializeDisplay( ) method to initializethe display 106, and wait for input from the user 244 or thecommunicator 210. If an input is received, the Main( ) method invokesthe MakeConnection( ) method 222 to cause a connection to be establishedbetween the local participants and the remote participants.

The StartConference( ) method 224 can be initialized by either a localuser or by the Communicator 210. StartConference( ) 224 invokes aStartCapture(n) method for each video camera n, and wait until aCamReady flag is received from each video camera n indicating that thevideo camera n is ready. The StartConference( ) method 224 invokes theDisplayDefault( ) method 234 to cause images from the video camera to beshown on the display 106. The StartConference( ) method 224 invokes theListen(m) method 228 for each microphone m to listen to audio signalsfrom the microphone m.

The StopConference( ) method 246 terminates the Listen(i) method 228 foreach microphone i, invokes StopCapture(j) method 248 for each camera j,and invokes StopDisplay(k) method 250 to stop images from the camera kfrom being shown on the display 106.

The methods associated with the Auditor object 204 include the Listen( )method 228. The Listen(mic) method 228 starts receiving audio dataassociated with the microphone mic through the audio card 160. TheListen( ) method 228 reads segments of the audio data based on a slidingtime window from the memory 154 and determines whether the audio signalis noise or speech by checking for correlation. If the audio signal isnot noise, the begin time t of the speech is determined. The Listen( )method 228 then invokes the AcknowledgeTalker(t) method 230 to cause anew speaker to be added to the talker map.

The methods associated with the Talker object 206 include theAcknowledgeTalker(t) method 230.

Referring to FIG. 7, the AcknowledgeTalker(t) method 230 implements aprocess 260 to determine the location of the audio source (e.g.,location of the speaker). The AcknowledgeTalker( ) method 230 analyzesaudio signals from different microphones 113 and determines whetherthere is a correlation between the audio signals from the differentmicrophones 113. Because the distances between the speaker and thevarious microphones 113 may be different, the same audio signal may bepicked up by different microphones 113 at different times. A time delayestimation technique can be used to determine the location of thespeaker.

The process 260 finds the onset detected by the first microphone 113 a(262) at time t1. To determine whether there is correlation between theaudio signal from the first microphone 113 a and the audio signal fromthe second microphone 113 b, the process 260 reads a segment of theaudio signal from a first microphone 113 a starting at t1, and a segmentof the audio signal from the second microphone 113 b starting at t2using a sliding time window, and calculates a correlation coefficient rof the two segments of audio signals (266). The sliding time window isadjusted, e.g., by incrementing t2 (270), until a correlation is found,in which the correlation coefficient r is greater than a threshold(268). For example, if an audio segment starting at t1 from the firstmicrophone 113 a correlates to an audio segment starting from time t2from the second microphone 113 b, the process 260 determines that thetime t2 associated with the second microphone 113 b has been found(272).

The process above is repeated to find the correlation between the audiosignal from the first microphone 113 a and the audio signal from thethird microphone 113 c (274). The location (x, y) of the audio source(i.e., the speaker) is determined using geometric and trigonometryformulas (276). The process 260 records time points t1, t2, and t3 invariables and invoke the AddTalker( ) method 232 to add the speaker tothe talker map. The process 260 continuously check the correlationsamong the audio signals from the microphones 1, 2, and 3. When the audiosignals no longer correlate to one another, the process 260 invokes theRemoveTalker( ) method 238 to remove the speaker from the talker map.

The methods associated with the Meeting Focus object 208 include theAddTalker(Location) method 232 and the RemoveTalker(Location) method238.

Referring to FIG. 8A, the AddTalker(Location) method 232 implements aprocess 290 to add a speaker to the talker map. The process 290 recordsthe location of the speaker in the talker map (292). A source map 340(FIG. 10A) is read to determine if there are other speakers (294). Thesource map 340 includes a log of all the people who have spoken. Thetalker map 350 includes a list of the people who are currently activelyparticipating in a conversation. The process 290 determines if aconversation or discussion is taking place (296).

If there is only one active speaker, which indicates no conversation istaking place, the process 290 selects a camera 103 and determines thecentroid and percentage of window to show the speaker at the location(300). If there are more than one active speaker, which indicates that aconversation is taking place, the process 290 reads the location(s) ofthe other participant(s), selects a camera 103, and determines thecentroid and percentage of window to best show images that include thelocations of all the active speakers (302). The process 290 invokes theDisplayScene(cam, center, size) method 236 to cause the image to beshown on the conference display 106 (304).

Referring to FIG. 8B, the RemoveTalker(location) method 238 implements aprocess 310 that includes re-selecting the camera 103 and recalculatingthe center and size of image based on the locations of the remainingactive speakers (312). The process 310 updates the talker map (314) andinvokes the DisplayScene( ) method 236 to cause an updated image to beshown on the conference display 106 (316).

The methods associated with the Communicator 210 include theMakeConnection( ) method 222 and the BreakConnection( ) method 242. TheMakeConnection( ) method 222 checks to see if a connection isestablished with a remote site. If there is a connection, theMakeConnection( ) method 222 invokes the StartConference( ) method 224.If there is no connection, the MakeConnection( ) method 222 checks theconnection again after a period of time.

The BreakConnection( ) method 242 invokes the StopConference( ) method246 to cause the video conference to stop.

The methods associated with the Display 212 include theDisplayScene(Cam, (x,y), percentage of display) method 236 and theStopDisplay( ) method 250. The DisplayScene( ) method 236 reads datafrom memory that is written by the video input daemon 243. TheDisplayScene( ) method 236 selects a camera, determines the dimensionsof the digital zoom, and implements the calculated dimensions. A filteris used to smooth out the images to reduce block artifacts in theimages.

The StopDisplay( ) method 250 ends the StreamOut(Addr) method 240 tostop the video output daemon 242 form streaming video to the memory 154.

Referring to FIG. 9, in some implementations, the cameras 103 do notchange viewing directions, and zooming and panning are achieved bydigital cropping and enlargement of portions of the images. In thiscase, a camera 103 can be selected from among the plurality of cameras103 in the camera assembly 102 by choosing the camera with the optimalviewing angle. In some implementations, the camera having the optimalviewing angle can be selected by finding the camera with the smallestview offset angle. In this case, the speaker will be closer to thecenter of the image captured by the camera.

In the example of FIG. 9, a first camera 103 a and a second camera 103 bare facing each other, so a viewing direction 322 of the first camera103 a aligns with a viewing direction 324 of the second camera 103 b.The viewing directions of the cameras can be different. A speaker 320 atlocation P has a view offset angle ViewOff1 with respect to the viewdirection 322 of the first camera 103 a and a view offset angle ViewOff2with respect to the view direction 324 of the second camera 103 a. Inthis example, because ViewOff1 is smaller than ViewOff2, the firstcamera 103 a is selected as the camera 103 for capturing images of thespeaker 320.

In some implementations, digital zooming is used in which images fromthe video camera 103 are cropped and enlarged to achieve a zoom effect.When there is a single speaker, the cropped image has the speaker at thecenter of the image. The size of the cropped image frame is adjusted(e.g., enlarged) to fit the correct zoom factor. For example, if thezoom factor is 2×, the cropped image frame has a width and length thatis one-half of the original image, so that when the cropped image isenlarged by 2×, the enlarged image has the same size as the originalimage, thereby achieving digital zooming. The position of the croppedimage is selected to accurately display the chosen person, e.g., so thatthe speaker is at the middle of the cropped image.

Referring to FIG. 10A, a source map 340 is stored in the memory 154 tomanage information about a continuous list of talkers. The source map340 keeps track of past talkers and allows the Talker object 206 todetermine the current active participants of a conversation so that thecorrect participants can be shown on the display 106 and to the remoteparticipants. To establish the source map 340, a SourceMapEntry class isused. A SourceMapEntry object includes the (x, y) coordinates of thespeaker's location and the times and that the speaker starts or stopstalking

The action, angle1, angle2, time, and (x,y) coordinates are componentsof the SourceMapEntry class. The action parameter can be “stop” or“start” (indicating whether the source has stopped or started talking)The angle n (n=1, 2, 3, . . . ) parameter represents the offset anglefor the specific camera n to the talker (the angle that the optical axisof the camera n would need to turn to be pointing toward the talker). Inthis example, two cameras 103 were used, so there were two angleparameters for each SourceMapEntry. If more cameras 103 are used, moreangle parameters can be used for the additional cameras.

Referring to FIG. 10B, a conversation recognition algorithm is used todetermine the participants of an active conversation and to establish atalker map 350 including a list of the participants of an activeconversation. There can be various conversational dynamics that couldhappen during a conference. For example, one person can be making aspeech, two people can be talking to each other in a discussion, or oneperson can be the main speaker but taking questions from others. One wayto capture these scenarios is that every time a speaker starts talking,the system 100 checks to see who has talked within a certain time. Bydefault, the camera 103 that is the best choice for the new speaker willbe chosen because he or she will be the one speaking and it is mostimportant to see his or her motions. An appropriate zoom is chosen basedon an evaluation of recent speakers in the source map, specificallydetermining the leftmost and rightmost past speakers in relation to thechosen camera.

In some implementations, if a speaker talks for more than a certainlength of time, and no other person has spoken during that period, thesystem 100 resets the scene and focuses on the sole speaker.

Referring to FIG. 11, a conversation sensitive zoom process 360 is usedto select a camera 103 and control the viewing angle and zoom factor ofthe selected camera 103 to ensure that the leftmost and rightmost activespeakers are included in the images displayed. The conversationsensitive zoom process 360 includes the following steps:

Step 1 (362): Select the camera that can best capture images of thecurrent speaker.

Step 2 (364): Read the entries in the source map 340 (e.g., by callingthe SourceMapEntry) and identify the entries having time stamps that arerecent within a predetermined time period.

Step 3 (366): Determine the leftmost speaker and the rightmost speakerbased on the chosen camera and the particular recent entries from thesource map 340.

Step 4 (368): Calculate horizontal and vertical offsets as a fraction ofthe total field as signified by the left edge (leftmost), the right edge(rightmost) and the total field (total angle).

FIG. 12 is a diagram 370 showing an example of positions of active andinactive speakers, and the left and right display boundaries. A marginof a few degrees d of viewing angle is provided on the left side 376 andright side 378 of the display view boundary in order to show the torsosof the leftmost speaker 372 and the rightmost speaker 374. Note that thespeaker location identifier 112 determines the source of sound, which isthe location of the speaker's mouth, so some margin at the left andright of the image is used to ensure that the entire torsos of thespeakers are shown. The left and right side display view boundaries 376and 378 are within the camera view angle left boundary 380 and cameraview angle right boundary 382.

FIG. 13 shows a diagram for determining a horizontal position (xPos) anda vertical position (yPos) of a selected scene to achieve a particulardigital zoom effect. In step 1, the system 100 sets up desireddimensions (xDimension and yDimension) for the zoomed angle using theequations:

${xDimension} = {\frac{1024}{Length}\mspace{14mu} {and}}$${yDimension} = {\frac{768}{Hight}.}$

The equations above assumes that the display screen has a 1024×768 pixelresolution.

In step 2, the system 100 re-centers the window by calculating thehorizontal and vertical position offsets xPos and yPos based on zoomsize using the equations:

${xPos} = {\frac{{xDimension} - 1024}{- 2.0}\mspace{14mu} {and}}$${yPos} = {\frac{{yDimension} - 768}{- 2.0}.}$

In step 3, the horizontal and vertical offsets are adjusted by using theequations:

${xPos} = {\frac{{xDimension} - 1024}{- 2.0} - {{\left( {{HorizOffset} - {.5}} \right) \cdot {xDimension}}\mspace{14mu} {and}}}$${yPos} = {\frac{{yDimension} - 768}{- 2.0} - {\left( {{VertOffset} - {.5}} \right) \cdot {{yDimension}.}}}$

FIGS. 14A to FIG. 14D are examples of images of people talking in a roomin which the images were taken by cameras in an experimental setup thatimplements the conversation sensitive zoom process 360 described above.In the experimental setup, a speaker location detector 112 was not used.Instead, a video of the people was analyzed to determine when someonestarts speaking and the speaker ceases speaking, and entries wereentered into a source map 340. The source map 340 was used to determinea talker map 350, which was used to control the selection of camera 103and zooming of the images in the videos taken by the selected camera103.

FIG. 14A shows an image taken by one of the cameras 103 with the widestview angle available (the least amount of zooming was used). FIG. 14Bshows an image in which a single speaker was detected, and the image waszoomed in to the single speaker. FIG. 14C shows an image in which twopersons are engaged in a conversation, and the image was zoomed in toshow both speakers. FIG. 14D shows an image in which three people areengaged in a conversation. The image in FIG. 14D was taken by a camerato best capture the view of the speaker who is currently speaking

The images shown in FIGS. 14B and 14C are somewhat grainy. The qualityof the images can be improved by using video cameras having higher pixelresolutions. Alternatively, optical zoom can also be used to improveimage quality.

Various modifications can be made to the system 100. In some examples, adelayed automatic zooming is used so that when a speaker ceases talking,the camera zoom does not change immediately. Rather, the scene changesafter a few seconds to provide come sense of continuity to the viewer.This way, if a speaker pauses for a few seconds and resumes talking, thecamera view angle will not change.

In some examples, the system 100 provides an advanced features sectionin the graphical user interface to allow users to configure the behaviorof the system to suit their preferences. For example, the user canadjust the duration of time that the system 100 will consider a speakerto be “active” if the speaker has spoken within the duration of time,the zoom factors for various settings, the margins d at the left andright image borders, and the time delay between detection of changes inthe talker map and adjustment of the display view.

In some examples, when the camera viewing angle or zoom factor arechanged, there is a smooth transition from one scene to the next. Thevideo appears to have continuous panning and zooming when speakers joinin or drop out of a conversation. This prevents distracting quickflashes.

Other implementations and applications are also within the scope of thefollowing claims. For example, the model numbers and parameters of thecomponents in the video conferencing systems can be different from thosedescribed above. The video camera can be color or gray scale, it caninclude, e.g., a charge-coupled device (CCD) or a CMOS image sensor. Thecameras 103 can be controlled using multiple embedded microcontrollersor a centralized computer. The system 100 can provide a GUI to allowusers to adjust parameter values inside of text boxes. A selfcalibration system that used beeps from the cameras to calibrate themicrophone array can be used. The camera viewing angle can be adjustedmechanically and optical zooming can be used.

More than one camera can be selected. For example, in a large conferenceroom where speakers engaged in a discussion may be seated far apart, itmay not be practical to show an image that includes all of the activeparticipants of the discussion. Two or more cameras can be selected toshow clusters of active participants, and the images can be shown onmultiple windows. For example, if a first group of two people at one endof a long conference table and a second group of three people at theother end of the long conference table were engaged in a discussion, avideo showing the first group of two people can be shown in one window,and another video showing the second group of three people can be shownin another window.

What is claimed is:
 1. A method of conducting a video conference,comprising: using at least one camera to capture images of people at afirst location participating in a video conference; identifying one ormore active speakers among the people at the location; automaticallyselecting one of the at least one camera based on a position orpositions of the one or more active speakers; providing images from theselected camera to a person at a second location participating the videoconference; and maintaining a database of one or more active speakers,adding a person who starts to speak to the database, and removing aperson who has not spoken for a predetermined period of time from thedatabase.
 2. The method of claim 1, comprising optically or digitallyzooming the images based on the one or more active speakers.
 3. Themethod of claim 1 wherein identifying one or more active speakerscomprises identifying people who have spoken within a predeterminedperiod of time.
 4. The method of claim 1, comprising providing a userinterface to allow adjustment of the duration of the predeterminedperiod of time.
 5. A method comprising: using at least one camera tocapture images of people at a location; identifying one or more activespeakers at the location; automatically selecting one of at least onecamera based on a position or positions of the one or more activespeakers; providing images from the selected camera; and maintaining adatabase of one or more active speakers, adding a person who starts tospeak to the database, and removing a person who has not spoken for apredetermined period of time from the database.
 6. The method of claim 5wherein identifying one or more active speakers comprises identifyingpeople who have spoken within a predetermined period of time.
 7. Themethod of claim 5, comprising periodically updating the database andautomatically adjusting the selection of camera based on the updateddatabase.
 8. The method of claim 5 wherein maintaining the database ofone or more active speakers comprises storing information about wheneach speaker starts and ends speaking
 9. The method of claim 6 whereinmaintaining the database of one or more active speakers comprisesstoring information about a coordinate of each active speaker.
 10. Themethod of claim 5 wherein selecting one of at least two camerascomprises selecting one of the cameras having a smallest view offsetangle with respect to the one or more active speakers.
 11. The method ofclaim 5, comprising sending the images to a remote party who isconducting a video conference with the people at the location.
 12. Themethod of claim 5, comprising determining the position or positions ofthe one or more active speakers.
 13. The method of claim 12 whereindetermining positions of the active speakers comprises determiningpositions of the active speakers by triangulation.
 14. The method ofclaim 13 wherein determining positions of the active speakers bytriangulation comprises triangulation based on signals from a microphonearray.
 15. The method of claim 5, comprising automatically zooming thecamera to more clearly show the one or more active speakers.
 16. Themethod of claim 15 comprising determining a zoom value based on adistance or distances between the camera and the one or more activespeakers.
 17. The method of claim 16 wherein determining the zoom valuecomprises determining a zoom value to provide a first margin between areference point of a left-most active speaker and a left border of theimage, and a second margin between a reference point of a right-mostactive speaker and a right border of the image.
 18. The method of claim15 comprising determining a zoom value based on a distance between thecamera and a closest of the one or more active speakers.
 19. The methodof claim 5, comprising automatically adjusting a viewing angle of thecamera to more clearly show the one or more active speakers.
 20. Anapparatus comprising: at least one camera to capture images of people ata location; a data processor to select one of the at least one camerabased on a position or positions of one or more active speakers at thelocation and provide images from the selected camera to show the one ormore active speakers; and a storage to store coordinates of the activespeakers in the room, and time points when each active speaker startedand ended talking
 21. The apparatus of claim 20, comprising a speakeridentifier to identify active speakers in the room.
 22. The apparatus ofclaim 21 wherein the speaker identifier identifies active speakers inthe room by identifying one or more people who have spoken within apredetermined period of time.
 23. The apparatus of claim 21 wherein thespeaker identifier comprises a microphone array that enablesdetermination of positions of the active speakers by triangulation. 24.The apparatus of claim 20 wherein the data processor selects one of theat least one camera by selecting the camera having a smallest viewoffset angle with respect to the active speakers.
 25. The apparatus ofclaim 20 wherein the data processor executes a video conferencingprocess to send the image to a remote party who is video conferencingwith the people at the location.
 26. The apparatus of claim 20 whereinthe data processor controls a zoom factor and a viewing direction of thecamera such that the active speakers are within a viewing range of thecamera.
 27. A method comprising: using at least one camera to captureimages of people at a location; identifying one or more active speakersat the location; automatically selecting one of at least one camerabased on a position or positions of the one or more active speakers; andproviding images from the selected camera; wherein selecting one of atleast one camera comprises selecting a camera having a smallest viewoffset angle with respect to the one or more active speakers.
 28. Amethod comprising: using at least one camera to capture images of peopleat a location; identifying one or more active speakers at the location;automatically selecting one of at least one camera based on a positionor positions of the one or more active speakers; providing images fromthe selected camera; automatically zooming the camera to more clearlyshow the one or more active speakers; and determining a zoom value basedon a distance or distances between the camera and the one or more activespeakers.
 29. The method of claim 28 wherein determining the zoom valuecomprises determining a zoom value to provide a first margin between areference point of a left-most active speaker and a left border of theimage, and a second margin between a reference point of a right-mostactive speaker and a right border of the image.
 30. A method comprising:using at least one camera to capture images of people at a location;identifying one or more active speakers at the location; automaticallyselecting one of at least one camera based on a position or positions ofthe one or more active speakers; providing images from the selectedcamera; automatically zooming the camera to more clearly show the one ormore active speakers; and determining a zoom value based on a distancebetween the camera and a closest of the one or more active speakers. 31.A method comprising: using at least one camera to capture images ofpeople at a location; identifying one or more active speakers at thelocation; automatically selecting one of at least one camera based on aposition or positions of the one or more active speakers; providingimages from the selected camera; and automatically adjusting a viewingangle of the camera to more clearly show the one or more activespeakers.
 32. An apparatus comprising: at least one camera to captureimages of people at a location; and a data processor to select one ofthe at least one camera based on a position or positions of one or moreactive speakers at the location and provide images from the selectedcamera to show the one or more active speakers; wherein the dataprocessor selects one of the at least one camera by selecting the camerahaving a smallest view offset angle with respect to the active speakers.